Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use db-ip lite database for countries #906

Merged
merged 3 commits into from
Apr 12, 2021
Merged

Use db-ip lite database for countries #906

merged 3 commits into from
Apr 12, 2021

Conversation

ukutaht
Copy link
Contributor

@ukutaht ukutaht commented Apr 2, 2021

Changes

See discussion in #219. With this PR, a basic lite IP-to-Country database is included automatically with no need for extra configuration. Unlike Maxmind there seem to be no restrictions around packaging the database as part of our self-hosted offering as long as credit is given. This will greatly simplify development setups and self-hosted configuration.

Plausible Cloud will keep using MaxMind's database temporarily so we don't have to put up a license notice. But I think it would be a good idea to move to db-ip using their commercial (more accurate) database.

Db-ip also has a free and paid city databases which is great for the future.

Tests

  • Automated tests have been added

Changelog

  • Entry has been added to changelog

Documentation

  • Docs have been updated

@bhavin192
Copy link
Contributor

bhavin192 commented Apr 5, 2021

This is awesome! I was wondering about following things:

  • How frequently we are planning to update the checked in mmdb binary file? (I think binary files from git take up more space, as each file is saved in Git even when we replace it). Git LFS is there but feels like too much of hassle for one small file.
    • We can add a step to download latest file as part of container build (I can help with that, if we decide to do so).
  • The self-hosted instances will get the new mmdb file whenever there is a new release, this means the file will be 5-6 months old before it gets updated. Not sure if it is a problem though. One can have similar mechanism as the Maxmind's container image, which downloads new file every month, if they want a latest file.
  • I was going through this thread https://www.mail-archive.com/[email protected]/msg37375.html But did not get a satisfactory answer on if we can distribute the database with AGPL / GPL code. There have been changes in CC-BY-4.0 which make it compatible with GPL, the mail thread does not cover that, 4.0 was not there yet at that time.

@ukutaht
Copy link
Contributor Author

ukutaht commented Apr 6, 2021

We can add a step to download latest file as part of container build (I can help with that, if we decide to do so).

I think that would be best. I was planning to update it manually every time we tag a new release but I think automatically downloading it each time Docker build runs is better.

Of course the ideal solution would be to download it in the background every month as it's updated. Even if we release new versions periodically it's not guaranteed that people will upgrade their plausible versions. I'm OK with having a manual update (like we do for User-Agent database and Referrer database) for now.

But did not get a satisfactory answer on if we can distribute the database with AGPL / GPL code.

I've emailed the Creative Commons organization asking for clarification about this. Let's see what they say.

@ukutaht
Copy link
Contributor Author

ukutaht commented Apr 12, 2021

License question is cleared up. Here's the response from db-ip people themselves:

Hi Uku,
Thank you for contacting us !
You are absolutely free to redistribute the DB-IP Lite databases along with your AGPLv3 project.

@bhavin192 happy to accept your contribution to download the DB whenever the Docker image is built. If you don't have time I can probably do it myself as well. Let me know.

@bhavin192
Copy link
Contributor

That's great! I will take care of the container image changes.

@ukutaht
Copy link
Contributor Author

ukutaht commented Apr 12, 2021

@bhavin192 I just added a task to download the latest DB from db-ip. I also rebased the history, the main point of this was to remove the actual file from the git history so it doesn't weight the repo down in the future. I'll merge this now and await for container updates from you. Thanks!

@ukutaht ukutaht merged commit 83a759c into master Apr 12, 2021
@ukutaht ukutaht deleted the dbip-database branch April 12, 2021 09:21
@ukutaht
Copy link
Contributor Author

ukutaht commented Apr 13, 2021

@bhavin192 sorry I'm in a bit of a rush to get this version out so I added a download step to Dockerfile myself: 4fd5cdf

@bhavin192
Copy link
Contributor

@ukutaht ah, no worries, I got stuck with something else, that's why I was not able to do it yesterday.

oliver-kriska pushed a commit to payout-one/analytics that referenced this pull request Apr 14, 2021
* Use DBIP database for countries

* Add fake data for geolocation tests
zdzichu added a commit to zdzichu/docs that referenced this pull request May 17, 2021
Since v1.3.0 (or, more specifically, plausible/analytics#906) Plausible ships its own country database.
Update the docs, leaving previous instructions example of getting another source of country data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants